{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow V2.0 图像识别教程\n",
    "\n",
    "教程参考官方专家高级教程：\n",
    "https://tensorflow.google.cn/tutorials/quickstart/advanced?hl=en\n",
    "\n",
    "这里以 TinyMind 《汉字书法识别》比赛数据为例，展示使用 Tensorflow V2.0 进行图像数据分类模型训练的整个流程。\n",
    "\n",
    "数据地址请参考:\n",
    "https://www.tinymind.cn/competitions/41#property_23\n",
    "\n",
    "或到这里下载：\n",
    "自由练习赛数据下载地址：\n",
    "训练集：链接: https://pan.baidu.com/s/1UxvN7nVpa0cuY1A-0B8gjg 密码: aujd\n",
    "\n",
    "测试集: https://pan.baidu.com/s/1tzMYlrNY4XeMadipLCPzTw 密码: 4y9k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据探索\n",
    "请参考官方的数据说明"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据处理\n",
    "\n",
    "竞赛中只有训练集 train 数据有准确的标签，因此这里只使用 train 数据即可，实际应用中，阶段 1、2 的榜单都需要使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据下载\n",
    "\n",
    "下载数据之后进行解压，得到 train 文件夹，里面有 100 个文件夹，每个文件夹名字即是各个汉字的标签。类似的数据集结构经常在分类任务中见到。可以使用下述命令验证一下每个文件夹下面文件的数量，看数据集是否符合竞赛数据描述：\n",
    "```sh\n",
    "for l in $(ls); do echo $l $(ls $l|wc -l); done\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 划分数据集\n",
    "\n",
    "因为这里只使用了 train 集，因此我们需要对已有数据集进行划分，供模型训练的时候做验证使用，也就是 validation 集的构建。\n",
    "> 一般认为，train 用来训练模型，validation 用来对模型进行验证以及超参数（ hyper parameter）调整，test 用来做模型的最终验证，我们所谓模型的性能，一般也是指 test 集上模型的性能指标。但是实际项目中，一般只有 train 集，同时没有可靠的 test 集来验证模型，因此一般将 train 集划分出一部分作为 validation，同时将 validation 上的模型性能作为最终模型性能指标。\n",
    "\n",
    "> 一般情况下，我们不严格区分 validation 和 test。\n",
    "\n",
    "这里将每个文件夹下面随机50个文件拿出来做 validation。\n",
    "\n",
    "```sh\n",
    "export train=train\n",
    "export val=validation\n",
    "\n",
    "for d in $(ls $train); do\n",
    "    mkdir -p $val/$d/\n",
    "    for f in $(ls train/$d | shuf | head -n 50 ); do\n",
    "        mv $train/$d/$f $val/$d/;\n",
    "    done;\n",
    "done\n",
    "```\n",
    "\n",
    "> 需要注意，这里的 validation 只间接通过超参数的调整参与了模型训练。因此有一定的数据浪费。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型训练代码-数据部分\n",
    "首先导入 TF 看一下版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.1.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练模型的时候，模型内部全部都是数字，没有任何可读性，而且这些数字也需要人为给予一些实际的意义，这里将 100 个汉字作为模型输出数字的文字表述。\n",
    "\n",
    "需要注意的是，因为模型训练往往是一个循环往复的过程，因此一个稳定的文字标签是很有必要的，这里利用相关 python 代码在首次运行的时候生成了一个标签文件，后续检测到这个标签文件，则直接调用即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if os.path.exists(\"labels.txt\"):\n",
    "    with open(\"labels.txt\") as inf:\n",
    "        classes = [l.strip() for l in inf]\n",
    "else:\n",
    "    classes = os.listdir(\"worddata/train/\")\n",
    "    with open(\"labels.txt\", \"w\") as of:\n",
    "        of.write(\"\\r\\n\".join(classes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相比于 TF V1.x，V2.x 中一个比较大的变化就是数据集读取和处理的工具更加简单，（虽然效率其实低了一些，但是考虑数据读取在一定的优化下，很少成为模型训练的瓶颈，这一点性能损失带来巨大的便利性，还是值得的）。\n",
    "\n",
    "TF V2.x中提供了直接从目录中读取数据并进行训练的 API 这里使用的API如下。\n",
    "\n",
    "这里使用了两个数据集，分别代表 train、validation。\n",
    "\n",
    "需要注意的是，由于 数据中，使用的图像数据集，其数值在（0， 255）之间，不适合直接输入模型进行训练，因此这里使用 rescale 对数据进行缩放。同时，train 数据集做了一定的数据预处理（旋转、明暗度），用于进行数据增广，而 validation则不需要做类似的变换。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_gen_train = tf.keras.preprocessing.image.ImageDataGenerator(\n",
    "    rescale=1.0 / 255.0, rotation_range=15, brightness_range=(0.5, 1.0)\n",
    ")\n",
    "img_gen_val = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1.0 / 255.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从 API 的名称和参数可以看出，这个 API 并不直接从目录中读取数据，我们实际使用的时候，要使用这个 API 的一个封装。\n",
    "\n",
    "在这个封装中，我们指定了图像最终的大小（target_size），颜色模式（color_mode），批量大小（batch_size），同时还有一个非常重要的标签（classes）。\n",
    "\n",
    "需要注意的是，这里的 color_mode 使用的是灰度模式，读取出来的数据只有一个颜色通道，因为书法的汉字全部是水墨，无所谓颜色。而 classes的使用，保证每次模型训练都使用统一的标签，如果不指定，那么这个 API 会按照一个内置的规则对标签进行编号，这个编号在不同的系统平台之间可能是不一致的。\n",
    "\n",
    "这里还需要注意一点的是，train 集我们对数据进行了随机打乱 (shuffle)， 而 validation 则没有。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n"
     ]
    }
   ],
   "source": [
    "batch_size = 32\n",
    "img_train = img_gen_train.flow_from_directory(\n",
    "    \"worddata/train/\",\n",
    "    target_size=(128, 128),\n",
    "    color_mode=\"grayscale\",\n",
    "    classes=classes,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=True,\n",
    ")\n",
    "img_val = img_gen_train.flow_from_directory(\n",
    "    \"worddata/validation/\",\n",
    "    target_size=(128, 128),\n",
    "    color_mode=\"grayscale\",\n",
    "    classes=classes,\n",
    "    batch_size=batch_size,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到这里，这两个数据集就可以使用了，正式模型训练之前，我们可以先来看看这个数据集是怎么读取数据的，读取出来的数据又是设么样子的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((32, 128, 128, 1), (32, 100))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imgs, labels = next(img_train)\n",
    "# 因为是 generator 所以可以用next来读取\n",
    "imgs.shape, labels.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到数据是（batch, width, height, channel）, 因为这里是灰度图像，因此 channel 是 1。\n",
    "\n",
    "> 需要注意，pyTorch、mxnet使用的数据 layout 与Tensorflow 不同，因此数据也有一些不同的处理方式。\n",
    "\n",
    "把图片打印出来看看，看看数据和标签之间是否匹配\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'益'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAD7CAYAAABqkiE2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO2debRcVZX/P6eGN2SAhMGIAQ2sRmhAwRAm+SEKIoJpEKExoAJqLxoH2lZ/LdBtLwZBEWj5gS0oCoITIA0NCHRLkKllNAFaZgljggwBJCHJG+pVnd8ft/apXeeeqvdqeO/VS53vWm9VvXtv3Xvuueee/d3D2dtYa4mIiOheZCa7AREREZOLOAlERHQ54iQQEdHliJNARESXI04CERFdjjgJRER0OcZtEjDGfNQY86QxZpkx5sTxuk5ERERrMOMRJ2CMyQJ/AvYDVgB/AI6w1j7W9otFRES0hNw4nXdXYJm19hkAY8wVwMFAcBLYZJNN7Lx588apKesn9ORtjGn4N/7/mUym5nFjPX9EZ2Pp0qWvWWs39beP1yQwF1iu/l8B7KYPMMYcCxwL8M53vpMlS5aMU1PWLxSLRQBKpZLbJi9wNpt12+VFzmaz7nfyW3mpR0ZG3HH9/f2pawhyufEaJhETCWPM86Htk2YYtNZeZK1dYK1dsOmmqckpQsFa615iay3WWjKZjPuTbYVCIbVf/tcwxmCMqTqHoFAouGtls1k3iUSsvxivKf5FYAv1/+blbRFNQl5kLeFFoov0LxaLKSagGYMv0Y0xTuoPDAwACauQ46Ia0B0YLybwB2BrY8yWxpgeYBFw/ThdKyIiogWMCxOw1o4YY74M/BbIApdYax8dj2t1A0qlEnfffTcAv/nNb4BE6g8ODrr9AENDQ+y2W2J6OfroowHo6elx5/EluzEmZTvIZDJRBegyjJvFx1p7E3DTeJ0/IiKiPYhm3ymAhx56iLPOOguA3/72t0Ai/YUBiIQ3xnD77bcDcNNNN7njAIaHhxkZGQES4x/AAQccwIknJnFcYhwcHh5maGgIgBkzZtRsUyi+RNoR2rdq1SoAnn32WdcOuc7Q0JBrk+wbHh4GYIsttmDBggWp88s1Qtsik2kMMWw4IqLLEZnABCDku/e3af1cpNsjjzwCwHe+8x1uvvlmoOLD19JWew6ee+45APcpUjGbzTrpKlL/T3/6k7MrnHbaaUBiQ8jn81XX0tJW2q3jDwTyOzmmVCo5r8MFF1wAwDe/+U13vLRDn0O2yTU/8IEPcPrppwOwyy67uGNCnpHIAJrDuIQNN4oFCxbY9TlYSCiuDNienp5U0I/49KHyAvzqV78C4Kijjmq5DaFJxhjDhhtuCMDcuXMB2G+//TjnnHOqfqtffGm33FMmk3Evv294XLduHT/84Q8BOOGEE4DETSkTj0DHKWiXJiQTy5577gnAJz7xCQC23nprPvKRjwDVE2AMaqoPY8xSa+0Cf3tUByIiuhxx6pwAaEouEOmn6a9ItSeeeAKAG2+8sW1tsNa66+tAonXr1gGwcuVKAN566y2332+jPodsGxkZcWqGMAIx+P3sZz/j61//elU7RkZGUpRfIhj9a8mnuEd///vfA7Djjjs6JnLggQdW3aM+b8TYEJlARESXIzKBCURI/9cGQtFpH3/8cQCuuuoqd7yvKzcD/5rawPbaa68B8Prrr6eMbiKdS6WSc+VJW/P5vNv21ltvARUGc/zxxzNt2jQAxzigIqll0dLQ0JCzNfT19QHVC6W0LQXg4Ycf5qSTTgJg7dq1ABx88MH09vY23TfdjDgJTCA01fVftKGhIeeXlxeh3ZBrCm0vFAqOygvNHxwcdFRbjpPJI2TFl8VKANOnTwdg1qxZQHJv/opE2Q6wZs0a1y45hx8voPdJW4vFIo8+mgSgirdhaGiIRYsWAdVRkp2Aesu+Q14e3be+h0YLEP3bVjwjUR2IiOhyRCYwAfAj6XR0m8z6uVyOFStWAHD99claq3aoAAKtUuhIQ5H22r0m30UKiZTJ5/PutyKx8/l8lZHQP5cYCUPGRTle2qLPq4/3z2uMce149tlnAfjWt77ltolLVdov96jvfWRkJCU9R0ZGWnYzanen/9xLpVLqmnos+O0dGRlx/aefk8+ujDGRCURERDSPyAQCqBcXP9pvtPT0JYE2dsk+kTwrV67k3HPPBeBHP/pRq7eQQiaTqZK8Al/y3nfffRxzzDEAXHbZZcHz6E9t3xBpJAY6zWT0d78dWtrrbf75fcahz/v8889z6qmnAjh7gbZ3yPe3v/3tQBK85EveXC5XJcmbgbRtZGTEGT5l29q1a901JWBqaGjI3Z9Ifdk3ODjoDKvbb789UJ0zQjOcVjAlJwFN6fxQVW048R9osVgM0it/AGrK5VumoTJANU31H25/f38qLFafS87x6quvAnD++efz/e9/v6ptcv52RHXql1AGXV9fn2uvhPcODAw474Ts04bKeouEBO2w0utJRi+CqoVSqeRCpWUylUmpVCq59m600UYAPP30085jIZ/aUCqTo/RBoVBw22QMDQ8Pu23yO704Su5B+tYY4/pS2qPvSQSCbMvlcmy77bYAXHjhhQDsvvvu7jhtPI3qQERERNOYkkxA00PfhVIoFKoSaUJ1JFnI/eYbpbLZbGqm1u4y8U0/9NBDAGy44YbueJn1C4VCit5pxiHnFal79tlnu3MI6/AZRyvQKoicf3Bw0G0T92SxWKwy+jWDdtDUkDtSEFKnCoVCauGTXlwkkHiIn//856kIw5A7s1HoGAhtUIWEUYlbVEOYk4wTud8ZM2a48fGpT30KSBij5OTcddddW24vRCYQEdH1mJJMQENmb5EIWor7sfI6Rl0zCJmBJdhFp+6SWVpi61etWsVjjyXlE4488kggkf5iwBGWoNcC+HYLbRMQqatdRb5ECNktWoHW66WPpN3aSNesntkOJlCvDkJoW8gN6Cdd0b9rh9QPQc6/bt06NybFQCgRlRqhVZXS7jfffNOd44UXXgCSlZSypPrWW28FkvtrxbUZmUBERJdjSjOB0Aw4MDDg9P7Vq1cDuDXzOvZdW/uFAci+l19+mTlz5gBJnDpU9K9cLlflnRBoBiDHyfnkU1iF2A10O0KW71bdVT7k+tr6rC3Mgj/96U9AklsAYPHixQ1dpx1hu/JcpZ5CLei1DLXY0mTlzJD2hBiAfwykx0KIUWUyGWdrahem9CQA1W4gqPb1vvnmmwAuA6/Osye/e/PNN1NRcJqeyWAczU3lZ9oJDdzQIK1H89tNXf3kJvl8Pmg8FSPh008/3dR12uEirOcODKGd6tJkQ6/FkDEsyGazzq1cK5lLw9dr6dcRERFTHlOaCYRWZ7355psccsgh7jtU6G0zGKuEGYu0rreirtlzNgKdEgzC0tZa6/pN1KRG0a5Itm6Dv0x77dq1qSQrmUyGvffeG6g2bkfDYERERNOY0kzAGOOMbGIEXLhwIQ8++CDQ3lV46wP8cFMdNqzhuyobRaet558q8JO+FItFNthgA6AS2tzf3+8S0LYLTU8CxpgtgJ8BcwALXGStPc8YsxFwJTAPeA443Fr7l9abWnVtIKFGMgl8/OMfB2Dp0qWOxorFXg/KRg1O6wtyuVzKgKT7QscEzJw5E6DpcvFxEmgOvnFZj2+dgt2PkpVjm0Ur6sAI8HVr7XbA7sCXjDHbAScCv7PWbg38rvx/REREh6JpJmCtfQl4qfz9LWPM48Bc4GDgg+XDLgNuB05oqZU1MDQ0xGGHHQbA/fff77bL7KnTaHU7dBowQWg9QU9Pj2MAm2++eVPXEhfh5ptv7hKlRIwOnwn09va6sSxqwdVXX+1UA+lnnYKtGbTFMGiMmQe8D7gPmFOeIABeJlEXQr851hizxBizREJyIyIiJh4tVyAyxswA7gDOsNZeY4x501o7S+3/i7V2dr1zNFuBaHh42MXsh9xpISbgG8e6BaEElT09Pc4uIHaUTCaTClDRumcjgSk6nZaWchLRGTI86szG3QY/ME33gYzlNWvWpArGjvWZ1KpA1JJ3wBiTB64Gfmmtvaa8+RVjzGbW2peMMZsBr7ZyjXoYLRV3SA0I5bfrBpRKJWfwkzBWbRgUimmtrUqMotFoZFomk3Eh21KVuLe3N/XMdFKMbnsuGn6W51Kp5KJd9bZGX/7R0LQ6YJIWXAw8bq39ntp1PXB0+fvRwHXNNy8iImK80QoT2BP4DPCwMeah8rZ/Bs4Efm2M+TzwPHB4a02sjVKpVJVUQlDPINjuxRdTBblcrmrhEiSsSIxLUuJrgw02cNtaTS4CsNlmm1V9PvXUUynVrb+/3zGRsaQvW98h/b7TTjtxyy23ANXrBELs1y/t1gha8Q78Hqh1xX2bPW9ERMTEYkpHDOZyOWcYlIhBvfTUd7lA9xqedDUbweLFi5k9O7HZvuc97wGSPvN101Zwzz33ABV31kEHHcRtt90GhGsAjFeyj05AX1+f6wf5zGQyzlCq3bQAt912m7PLaBdgaOXnpLsIIyIipi6mNBPIZDJce+21QEWCfPzjH3d6USipY7cxAEGxWHRuwIsvvhiAPffc0/WVeAqstal8APX0zNHq7Il3QHD11Vdz6KGHVv22VCq54/x0cfl83rVHtvX19TlpKdsuuuiimsk7tJSs9/zbVfjVv670yzPPPOPqHsi2lStXujToulQ7VLOikNRvV9KZKT0JWGvZa6+9gEqH3XrrrSxcuBCouLjeeOMNoLrCbadAG8LGoqrUi3Po7e11hs8f/OAHAGyxxRZAQjGlPyRHnab7eu1AvZdeXjSZUHTtBz8OQRux5JwzZ850ZdZ0kZJQNKPfNl1bInRe3Q9yz/J7aaOoj5lMxvWHdr9JZJ5P2/P5fFXWYDm/PA+5lhxfLBZdm+Te1qxZk6r0vPHGG6eyO/ll4PT5dZ5MQUwqEhER0RKmNBOANOWaP38+N998M5BOv9TT0+MSMrz44ovuHDJ7ixTt7+9PudPqSWCdd1CnOZPvvrsrVFJax/H7+7RE0JTxlFNOASrut+nTp7v9woYksGQ01JMm4r7L5/PufNrVKvcpfSb3t2rVKl555RWgIoF1ijc5h45Q9GsGFAoFd145XmeD1uW8dHl1v41+vr9vf/vbTlILIygWi21L2aXrUwj6+/tT46RYLDopH2JDMjZDGaBDpdqaaXdkAhERXY4pzQR0UhHtcnnve98LVPIJiA6XyWS4++67gYqOtddee7Fs2bKq8xYKBbd/1qxkGYTM3L29ve5askJOr2EQ92SpVHLn8NN6aTYh37U+H8r378/w06ZNc2nTli9f7toh1//P//xPoCKdL7jgAubOnZs671igazoIpA/Wrl3r7kEMsWJk3HfffV3fitRds2ZNSucN2UC0dJR+EcmnU2lpd6bfv7rgqF/U9LXXXnMMQPatXr06lSFa7mV4eLjqu79NrqlZi9hNpP3XXXcdO+20U9U9Dw0NpZiRPr/cs7C9OXPmBAOqWkkv1vIConag2QVEUF28A5IHLg9fOlcbVfw8bqtXr04VMJ05c6braClbJRgZGXHn2HffJCZq2bJlKSqno+D8BTN6wMqD1+qDvGz10pDrTMECY0xqmxy/++67p9QePWnoYpuhwQ5w5513ss022wCVCXbfffd1xVhCC4JCHgN9D3L+WuNQF3EJQefg8ycX31AJVC1okuvrSaaWCjdWyLPW6klIhdQvcqiPBDKRiIq7yy67pAyl+r7qqQO1FhBFdSAiossxpdUBILV2QJfz8qVzaJb0/dgCkXQHHXQQUElaomffkLtJsG7dulSRUkHIVakNg6GIR79c+dDQkLv3UFkxP/JO1KBWsMsuu7j+ev3116vaA2kjrZas0j+aSenYBJ/i6nONRTpns1l3Xv85h9QHHUGpXbOtMmO9AtNnRrqEfb3SatKebDbrxqEUwVm1ahULFiTCfJNNNgGiizAiIqJFTHkmoHVqqM7B7pcGC+GAAw7gD3/4A1Bt2PL1Sz+QAyquM20w0+vy/eCfkGFQu3Z8iVev3blcLqXz6iQegnYHR0kWKG2/CCXB8CH3XGu1p8/o6p1L2wlCRUcF+hh/vz5/q3aAEHz2B7VzJfjXDd378ccfDyT9853vfAeAz372swDMnj3b9V8zUYRTfhLwLdeaWvoUHWDRokVAZWHLa6+9FvTByssvhhl5uUMPMpvNVr380i7/2NDDleNHRkZSA1X/3l8WXcuINZ5JOUKGv7Eu+Akt69aGz0byQIZe1lCIrWC0ibATjOOQjowMTVQAJ510ElBJ1LLNNtvw4Q9/GGguL2RUByIiuhwdxQSKxWJNepfNZlPGI70EVR//xS9+EagYw0RKDA0NOZ96KLlISJqEFiH5CLnympHIjUik9XnJbbdCnqlmtbp4rGwTnHnmme77jTfeCEQmEBER0QQ6hgn4bjM/FrpQKKSCLzKZTMqtdsIJJ/DrX/8aqMSkh7K2xjRWEZ0KHZ0qLFMHNPmRqKVSqaXaGpEJRER0OTqGCUAi2f0wYK3z+4EWmUzGraS78847AXjkkUec1VRmVL2u3E/YEJlARKdBQo9HRkZSNoFSqRS0QbVSY7NjJgHxq/tZU7R/Xm5U9hWLRZ5++mkA7rrrrqpzaXRrEdKIqQUZ6+KK1equn1fQRytZtKM6EBHR5egIJiDx/tr4JzOgjraTWfCCCy4AkmysDzzwgDuHPh5IxdtH6h/RyaiXfEagoyU1a27FZRyZQEREl6MjmACEk1RCdby9fH/wwQcB+M1vfpMKotBx4iEGIMfXS2gRETGZ0PYw/33QazWkVsQ222zjEso2g5YnAWNMFlgCvGitXWiM2RK4AtgYWAp8xlpb1zInaZQ1pZEXWPtCf/7znwPwP//zP0A4W66OsvInEn2+qBpEdBp8o7gWaDJei8Wi23bMMccAcNxxx7WUWagd6sBXgMfV/98FzrXW/hXwF+DzbbhGRETEOKHV0uSbAx8DzgC+Vq5UvA9wZPmQy4BTgAtHO1etBCB6hnv00UeBSm6/UGIIjdCS3MgAIjoV/urBnp4e5s2bB1TqR2SzWcd6ZZ/EwTSLVpnA/wO+AYhivTHwprVWVs+sAILZLY0xxxpjlhhjlsga9YiIiIlH00zAGLMQeNVau9QY88FGf2+tvQi4CJJEo7J2wGcAWopLEoXnn38egJtuuskdp3UmH9r4pxNT6s+I9Q+jMcB6CT7HCxINKJWO8vl8Vck1vS+TyXDccccBcPTRR7ttAl3QVackaxStqAN7AgcZYw4E+oANgPOAWcaYXJkNbA68WOccERERk4ymJwFr7UnASQBlJvB/rbWfMsZcBRxG4iE4GrhuLOeTGmt+eim9YkrSXcvnrbfeWjcdlV9au1AoTKpLsB0rFzWTGct5xlqMcywIJezUac7qBay84x3vAOBtb3ubk1ZSnejPf/5zKqtOO/MlhBjBYYcdxt/+7d8C1SXB/fLguhZhaJv0iV+70FrLd7/7XaBSFxIqWYFOOOEEt81/LnplrHYNQtIvPosNpS1vBG2pO6AmgYXGmK1IJoCNgAeBT1tr6wY267oDEucvN6UXTsjA00U35Lhf/OIXAFx55ZXBhUZyTj8lVysDu1Hoa4VeSH+SCKksoQEdcoX67W5lkOhYjHqFQiSdW6iI55e//GUA/v7v/9615bLLLgOSSsW+cUv+z+fz7gXQL5pflbivr89dS19fjvGLic6fP98VqamXx79ZaKN1vTySOkO0n4NS11LQY8B/BnpcNVN3oC3BQtba24Hby9+fAXZtx3kjIiLGH1O6ApFORCJpw5YvX+5YwaWXXgqE05GJJCkWiympWSwW3ezqS7dcLhekjCJp9D5/mz6nnwQ1m806g5AvhbSUk8/+/v4UBQ1JSF06W5fU9tuoJaWWwvr8xphUiW9NT/3ozRBr0dWGNHWuNQ6byZ7rX1N/1yxRB5jJtcZSyaceZDVfoVBIPWOdgbpeRmk/UY4+Xpe3C/VNrEAUERHRMKY0EygWi6kcA/l8nj//+c9ApY6glnwifbQOKt8lvbieTVsJx/QRMuRoFqLz8Ovjdby4v7rSvxe/GKd2HbVaqUZDt19y7Ouy7PLpu3qNMe6Z6YKhtSR+M0zAtwnpe9f6tm9z6enpabmPdFWlsQbx1LLfFIvFlOuvUCgEmVfILuRjXG0Ck4VMJlNlOISk47QlGqofsp+LTXsQQgNb6gkItfM9Dq1CU3//JdJGJD+/4vDwsJvQBDpbs091Q4VJGoUegHJtTU99dWBkZMQNcH1P9drmG3ObQajQSKiIjK8iFIvFltWBkCFbt8uf/DOZTErQ1DNG62P1ZOevO2gEUR2IiOhyTGkmYIxJuXcKhYKbDf3SVmvWrHEzsRjJxirddBmtZtmAVr10ijT5rBXNqEuO33vvvQBcc801VRFjkJS+8uMmhNHst99+LvqsWWSzWdcmrbJIX8m1Vq9eDSTGwxkzZlTd5+uvv+6MuLowqS6Xro8fHBx023Qx1ne/+90AfOhDH3LtEPiZqjVd1mpBKDlHq9C03E+Rp1GPleny6bXOL9doByITiIjockxpJqB1PZkVtXHHn+mnTZuWkgg6cYM2Gol0k6SPIrVmzJjRdMRfKGhEoCWHtEPKUd90002uuOpDDz0EwC233JLSc0MST66zZMkSrrnmGgD2339/AL7yla+4c2g3oEheuWdxB2p7ij7ed3fp2g7CCkT6X3/99Zx88slV9xlyzYUktu6z3XbbDYBDDz0UgK233pqFCxcCaQkq+Sp81HPPtYp2GmHHG1N6EtDqQL0HWo+WhQaHHkSiNshnrfM0Ah3foMNCZbAvW7YMgHPPPReAyy+/vG7Ib8gf7k9UK1as4KWXXgKStOyQROrJJHfAAQcAcOqpp6aiKbWV3bf2W2sdlZfPjTbaCEjUgx//+McAfP3rXweSSdo3cuqFY776lc/ng1mg7r//fgCWLl0KwHbbbed+c8ghh6SODxkjdZEP2edXkO4GdM+dRkREBDGl4wSmKorFoqPWmq1IDYVvf/vbQLIOAqqLULQCnxlp95R8vu1tb+MjH/kIAOeffz5Qkaj5fD6V+CKk2kj5t2uvvdYVh9Vsol7JLH/BTK3Crv7y20KhwI477gjA8ccfD8AnP/lJd2xo2bCcW/Zpv/5UovNjRYwYjIiICGJK2wSmKnS02gsvvAAkhj4x+klCVYEOMmkHc/Pdk1CR7E8//TQ//elPgYrL7yc/+QlQnbxCG2RFGl93XbJqXHRybZDTQTohJuC7R30prb9rfV7fw2OPPQZUSnbLdY466qiUfUPbk7S7U0cxtgu1VoH6+0Lb/AAyvS20rxlEJhAR0eWITGACITP8ypUrufvuu4FK8M8555xTM9lqaPVZK9fXFnI/Dh0qblFJ+hHKUyAW9ZGRESdJZZtuqx8COzg4WLWC02+HH+AVcnvKb+T6cpx8F+/KGWecAcADDzzgjpcgo3/4h39w15f2aObSah4J7WHSwWpyf9qjIoxL2iHh7rlczh0n61rkXnUb9fNpxqsRJ4EakAfnfw4ODqa2DQ0NVS0hhWRwyqCU9Qfif7///vs555xzgDAlF+hJoJ0ZkbSxzY8A1PC36QKZoeW3/jqOEKy1dQvEhtox1vyRArk/yUV54YUXunbLupIlS5aw9dZbA/DNb34TSF5SiYlo1UV42223sd9++wGVfpk1axYvv/xyzd/sumuShuPiiy8Gkr6QcSUu6h122CFlyNS1CJpBVAciIrocU54J6Nh7qA4uEQkis6nEsUM1nbzjjjuAikFrYGAgJQlkFeG6deuC9NSXYNls1kkVYQKhyLd6mMh8iCE1o5Z7DkZf0twJ0M/Jz2t4xRVXsOmmmwKVsnbynKB6nQIkLE4YjP701zXoaEt/bNZiAdJOCYDaaaedgGQsy3MR1vK9733PsYI99tgDiIbBiIiIFjHlmUA9Xcg3WA0MDLgw08MPP9z9XuLy5VOXQRdp/tZbbzXUnmKxyJo1a4CKhKwXJNOJEAPhWOHnN5hshPJIaOOoJJ25+eabgYrUbwcymYwz5ulx5a9J0W0S+AwC4KmnngJg4cKFbLXVVgCcd955AC64q1lM6UlAJ24QSrRs2TK+8IUvAIkVHipUbXBw0L2Yr776KpAMXKGFq1atcuf2X36d8rkeQrRaBpcMgHqGsU5Co5NWO7MwtQP1FmxpaFWy1YlaL2QT9ULG0tDQkHv5hdKXSqUxjQedMeqJJ54AkszNAN/97nedqnvggQc23uaGfxEREbFeobOm7gahDT7PPvsskND8P/7xj0BFKsusq+me/G5oaMjlJNSSWhiAbwDr6+tL0eSenp5UPvlisRhMSDKV0Cg97jTDoDyTWpI2tA6iXdfUY0QbHAWNqlohBvrcc88ByRqJbbfdFohMICIiogmsN5PAnDlzmDNnDl/5yldcbLmUcZbaArJd79MoFApOWst+ifOXv9AMPjw8XBUcpLcPDw+nzjVZ0IlLakHWKRhjGBwcbEhi9fb2dpxxEMbuau2EFbXNotFnpdGSOmCMmQX8BNgBsMDngCeBK4F5wHPA4dbav7RynVHaAFSWr37/+993+/zQUg1NEUODpNmw0dDvOk0NkD6bPn26M5QKrLXMnZtUk7/tttuqjh/tJek0daCb0MoYa5UJnAf8t7V2W2BH4HHgROB31tqtgd+V/4+IiOhQNM0EjDEbAh8AjgGw1g4Dw8aYg4EPlg+7jKRG4QnpM7QHvuHugQceqHvsVKZ8rcKvSTAwMBB0W8pxwgjGik5zEXYTJosJbAmsBH5qjHnQGPMTY8x0YI619qXyMS8Dc0I/NsYca4xZYoxZIv78iIiIiUcrU3cOmA8cb629zxhzHh71t9ZaY0xQ9FprLwIugiS9WAvtANpfGagboO0Xs2bNAuCuu+5ywS0+cxgNnWgUnMqoxVzleYg7sLe3l9mzZzd9nVaYwApghbX2vvL//0EyKbxijNkMoPz5agvXiIiIGGc0zQSstS8bY5YbY7ax1j4J7As8Vv47Gjiz/HldW1o6CqIUGh1+6HMmk3HhpnfddRcAf/3Xf+2OFykkHgSd2CIECcraZJNNgISdCUOTNGohZGS5NHUAACAASURBVDKZCV0xORHQUtxfQwCVUGKdst3vg/7+fpdwRFfJkvTwkoi2Va9Mq5ac44FfGmN6gGeAz5Kwi18bYz4PPA8c3uI1xoRuc0/pAaIhcQChl8rP3pPNZrn99tsB2GabbYBqFSG0XsKPqtPGwJkzZwKVNRtQiYyTpdj6HHpg+/fiV2geT4xHZmFN4+Xlnz59uusvvcbgXe96F1BZzyITcy6XY8MNNwQqz6C/v9+9/GLMndRJwFr7EJBKYUzCCiIiIqYA1hufjsyGmlr6FWZ0LLzMyDqQSEs+mY2FCteTsBMJncKrnntPoFew+Tnp7rjjDv7qr/6qapv+vUgwkfDNSExpr8S253I5Z4QUKZ/P51PqnKQBGx4edueQY3p6etw2+ezp6anarz/z+bz7LsfobfpTn6/W8bo9tY7PZrNu7MgqVR2tKczHGOPUKFnavPHGG7t9Ap19WX6rKyfF9GIRERFNY8ozAT9P/M0338yHP/xhoJK1VXQtPRPLzJrNZqsSgQhkFhddVq8OnMwwYGErg4ODweCcn/3sZ0BFSsgx/f397v7lnrbddtuqGouQrHiTexU21IqUkfY+/vjjQDin/sjIiJOksnqzHdeebIhBUCdzlf7QNhKBMAZhdjq9mF4XIH2jE6X4RWEb6bcpPQnoZBHSuR/60Ie45557gAoVnj9/PhCO69dpnbWKIC/MnDlJrNNf/pIsf+jp6UnRQr1Nf2oqqT97e3uD5/CPl/b09fW54+WlzeVyKYNQT08Phx12mOsbfY6Qr39kZMS9dELRpc/0OdrxIob83TLYc7lcKpOTP6inGjRFl3GXzWbduNKqqa8K6cndT5oTKpCiVbhm+iuqAxERXY4pzQSgIuF0WaoFCxZU7ZPyXrlczs268tnX1+dytkl6sXw+zxtvvFF1Xk3B/PLck4FSqRQ05vnGv1AbtUFOS36oZgxaZap1rkahJaQ25vrZfUN0uZFrTDaMMSlD3+DgoGNyob70a0DoZ6GNgP7+Vu83MoGIiC7HlGcC/oxaLBZTBjNxT9WKgV+xYgVQrbtLSihhACKpOiUoSTMBDd8WIJK1t7fXMQb9Oz8Tsi7j7Zct0+cXaENpvbz88rl69Wp3DmFeuly5n9V5tHz/+tr+cWJM0wlfdFWoH/zgB0DFBToyMpJieZo1+QZha22qpJp2M2sGANUVnGSs1Us0UyqVUq5sa22K/UYmEBER0RLWOybQ09Pjtvk6rfYOaJ1ZXDk64aRs05IgdD0IS0pdZNMvJqpLjWtIe319vlgsplhMqC6gLlwqEMmaz+dTpaxXr17tpOBBBx0EJKzITzAqGYbe9ra3sdtuuwHwyCOPAIkbbPXq1VXt0EzMrwY1Gcjlcql2bLDBBqkEoOvWrXMuVvGW6PLlH/vYx4AKc1y8eLHLuSDHSX9Pnz7dPR+xAxQKBddXwoKGhobcGJPALek/XWNQzp/JZNzx2kvQir3GdIIRZcGCBXbJkiVtOdeaNWucUcl30eh71XTMP25gYMBNAvU6V/YNDw+naGTIBeS7e/w2yvV9Sj88POxeYE0/pequlLcaGhpyk4BerALJ4BRaqq8jg/bRRx9NtVuw0UYbufbLINY0Vdobqugrhke59mREXPb09KQyDk+fPj1VHg5g332rI96lvatXr3ZZrOUlfOc73+nuWZ6t9Ofw8LDrB7l2rfUegsWLFwO49QKZTIYddtjBfYfqCa1Rg60xZqm1NhXmH9WBiIgux5RnAjrgBMLGHYG+Vy2VffVBZ+UVyails0+5dd15kfr3339/VeUjqMSLj4yMBI1dInVEmsh1Vq9enZIgAwMD/PSnP3XXlzb696LXF/jqQy6Xc8eJxB4YGKhJ3SVzs8Ymm2yScqfKNY0xbS3t1Sx6e3tT7dCRovLsCoVCqmissMrBwcFUv/T29rrnF+qzDTbYAKg8H8lyra8pWZ3lu5xXrnn55ZdXnSufz/O+970PwCUS0aphPUQmEBEREcSUNwxqww1US3Z/dhwaGkqtLDTGuN/6+h2kK8WE3FOFQoHXX3/dXR9g0aJFLljE1z0zmUzVajNpm8APmdUx59LGQqHg7BraaCht8vV0XZQztIJSmEbIRajdjaKvClt57bXX3PX9Sk/W2iojlz7nREDbKPz0c5ox+oZbSLtY+/v7U3aNoaEh9xx9RpDNZlP2k76+vqrVg/Lp97ceC0cccURVG0ulkkur/5nPfAZIWILPYPTxo7GE9WYSEIQovxwzMjKSKjlVLBZThkRNe/faay+gYg3Xx/mZjscKa60zPOpJwH9JQ4tt5Jr9/f0pQ6IxJnjP8vt68fh6gZQ/gcjgt9ZWUVu/bboKsGAy1QFpj14Lol8438oeuidBLTWp1v1ptcmPUYDGy5DJJDY8PMxXv/pV9x1g3rx57l723ntvoGLMhdFzREZ1ICKiyzHlmUC9JZQyGx955JFAspxVjpdVgYODgym3WrFYTLmUmikrXqtyTzabdeqDH0sAYVeilmpyvE/9dV67evS73lLo0DoEjdC2EI3tBISYVAhjYXITqcaErqWZjKgg3/jGN9w2GQO33norgFs/M1rZOYhMICKi69ExTEAMGL5hQ0v20Oop2S/Gt76+PidBjz32WABuuOEGoNrNEzKO1YNmAKGY+hBq7Q9dMySN6rUt1J6xXDui86Fdl8K8tBvRj3TUz/r6668HEjsBJPkwRkuC0zGTgG9kC9FOv/jo9OnTOeWUUwC49957gYS2i/Hq4YcfBghGaY315Y+ImGholc4P9dah4VrdlN+cffbZQJJcB5JcjaMJhKgORER0OTqCCUgBBu2j9mGMcexA/KT/+7//y+9//3ugsiDDGJPy2Yo7bnBw0J1D+68jdY7oJIg7UK/LqMdc8/m8G886hgGqYzVqITKBiIguR0cwAagO1NH/hwxmS5cuBRKDn+8CCQVGiCGlv7/fzYqhFWQR6xdquWghiRj1DWaabfrIZrNtS+IxGhrNZl0oFFK/ETacyWTG1yZgjPmqMeZRY8wjxpjLjTF9xpgtjTH3GWOWGWOuLJcoi4iI6FA0zQSMMXOBfwC2s9YOGGN+DSwCDgTOtdZeYYz5IfB54MLRzqdXdYWg0yppS6mfC6BYLLoVcaJHyay4bt26utIhYuzQORJ81qbZWydUbsrn86nQYL1SdCzpza21VeneIfE6+WHaE8UWRkO91Y0+WlUHckC/MaYATANeAvYBjizvvww4hTFMApIzz6dj2h0iOeH+8Ic/uP0hg0m9BBbj/XBCL4cg9JJMNchg14PMX4ijVbRa9HoiUCuJDCTPp16yGV8tHR4eTsX7a3edXo7sn3My0IhK0bQ6YK19ETgHeIHk5V8FLAXetNbKm7kCmBv6vTHmWGPMEmPMElltFxERMfFoRR2YDRwMbAm8CVwFfHSsv7fWXgRcBLDzzjtbSYzgU7PLLrsMSIpn3nXXXQC8+OKLdc890ZJ2tJyBsu5gsqVDO+AnvigUCik2Vmsp92RBL7vWy8VDz0PuRai/LgQq0MuF/XUTY4nVnwg0ssallRZ/GHjWWrvSWlsArgH2BGYZY2Ry2Ryo/8ZGRERMKlqxCbwA7G6MmQYMAPsCS4DbgMOAK4CjgetGO5ExxrlgfCPTfffdB8BVV12V0i/1TDyZerYO2wwZZCazgGm7Ic8gZIsRxqMLjXYCdH4D/Xw++9nPAnDAAQe443SyGcClcLv22muD9yO2A82MYPLG46677grA1ltvDYyNjTU9CVhr7zPG/AfwADACPEhC728ErjDGnF7edvEYzuWSWfhUUi/08aP9dHHGyYZO9gHVBS98D8ZUhh+ZViqVqvLfQXWSDnkpJjPbsLWWQw89FKhkE+7p6XHLbeWFCVWo/u1vfwuEJ3KdIbpTJvpPfvKTAOyzzz5j/k1L3gFr7cnAyd7mZ4BdWzlvRETExKEjIgaFAeiILT+3OqQTa3QKMpkMb3/724GkQAdUlxOXYhXPP/88UJsR+DEM4x3TUMuI5UtrWan5xS9+MeUrN8akXIRy/I9//OOOUNcKhYIrT/+5z30OqJbiAm2Y9l1/ofLfnWjobcQgKOgMU2ZERMSkoSOYAKSlkvyv9cxODbKx1vKP//iPAC4JpI5mvOSSSwD4zne+A1QYgRwHFeOo3ibQyT/HS/qE+vaEE04A4Etf+hKQFGf1y7Hp2HTJ23DSSScBcOqpp3LaaacB8O1vfxtoTlK1A37qM52eTRBadyIGv6mCZvq3YyYBgV84xK/62knQL6a8AHoyk0F2zDHHABWj4WmnncayZctGPe9437Nuq55c/KzBui6fn6Clv78/lSVZ2j84ONhwqazxwP7778/+++9f1Q5dCERQLBZTocSdUoV6rGjGQBnVgYiILkfHMQEfnUzHtPQUGqalt28wk0ISuVzO0eMnn3zSHV9rbYE2WLUb9SS0Ty2tte556JoE0jZhOtL+bDbr3IeS2GUyMhLPnz/fuQO1GuOroLouhbAEv2hJp6OZ/o1MICKiy9HxTGAq6GT5fD6V1imXy6V0ZGEJhxxyiCsn/q//+q9AdSlrgU4LNR5uUR2hWS+OXtq1bt065yL0S4/pc+hzffGLXwQqrOKf/umf2noPY4FmK6FqQ5qx+Uk8p8L404g2gYiIiIYRmUAbUCwWXVisJDQplUqpgKdQEUpd58/HWOsbtANaAkq7f/SjHwGVZ3D66aenVs3pYCG/vfo+N910UwC22247lzL+z3/+8/jeVBlaOoYCpLSdwLeRLFq0CEja+m//9m9A9fMYb9etHz5vrQ0mLpEx1oxHKU4CTUA6XLsvfRqmFxWFSqX5GWBDA2u8s9RoN5l+OcT4J+5AKdmmYwLk3kOGxZC//aijjgLg05/+tJtcvvzlL7vz+ll12/lyaQOnnuzGklFos802AyrrCyYaobHg5+OEigGzmViaqA5ERHQ5OoYJ1JqVpZLKRRdd5JKKSKKRyYJQLpGUvb29KUmWy+VqRqTpfHUhyj/eawV03j/5rtvqGyGlotPg4GCVIbAR6PsUV2KoHLpA5+xrNVK0HW7JyXBV67J8whx7enrcuJP+y+fz7h6lqG4jiEwgIqLL0TFMoBa22247ALbddlsntSabCfgYGhpyxi7RzQqFgpvF/Xh7Y8ykBKH4gTC9vb11Y80POeQQAM4//3ygtdRZ2vgmUlUbTOW7MBKdBqxVtGO9wmhVfNoJrfP7blrdH3Lc3Llz3SrJnXfeueHrdfwkoNGJRkKBPByfvkHlAWojz2Tei1a9NKWEarVA2ihRf63Qcj2YfUu2bsd4LD1uR8KPyYpclb7SWZ59o+kuu+zCpZdeCjQ3UUd1ICKiyzElmUCnJHWQ2XnDDTfkXe96F0CVgVAknW8EzGQyKSYwnusD9DU0hoaGnOQI+ZdDaweahY6e9N2SpVIpxQS0q7DVfgkZPUMRg7qOgM9IdOoxv9AIVJhUO1SPUPIc/Xz8vlq3bl1LTCUygYiILseUZAKdApmd3//+93PGGWcA1WWuBH7QjzFmUnRMX7rpik86nZZIIIlm1Ea9ZqFXSOqoSjl/qHqRtLnVvAqDg4Op52KtdW41v9aAtAlgyZIlQJJgxWcJ+h4ms+R9q+xjSk0C42FR15Z63Zm+yqEzHck2CYXdaKON3O9q0UyoDJR8Pj+pS1RD0WcyKUm1ZqjQ6FCEWqPQqkjo3v3+lvY04/f2oUvC6RfZV0tGRkaqsltDJUZi+fLlwRJ5Iphk7ExG8pRWPShRHYiI6HJMKSYwHuqAtdbN4jqiz6ePesaX3PW/+MUvgERq+dRP02r5rTbCdUKyCr0EWqh/JpNx24QViDScOXNm09eS/lm7di0vvPBCap+wDmEAIfdhs3jqqaf493//d6CytBnS42l4eNg9IxkLUszjwgsv5Atf+II7TtotfbXhhhsCsGrVqpba2gxajYiMTCAiosvR8UxA61jjbRisF8cvs3+pVHJ6qtQY0BJBu49qLbHNZrOTauSUPi0UCqnkGdou4uu5WmduFg888AB/93d/l9ruu92EGeTz+ZaDfZYvX84VV1wBVJhAyOCYz+er3JZQSZm2/fbbBzMWC3PxGcB4unz9NrZqGByVCRhjLjHGvGqMeURt28gYs9gY81T5c3Z5uzHGnG+MWWaM+aMxZn5LrYuIiBh3jGVavxT4d+BnatuJwO+stWcaY04s/38CcACwdflvN+DC8mdbMF5uNd8irOHP5n19fal4bu3iEqk1MjKScoVpN1InJFDVtgyRJvl83rXzscceAyoJUq+//vqmryX9M2vWrJTerVcz+s+gHSG/1lpn89AeDv85as+Pv6pVpx7T7k7fexEKv24FocA43406ODjYkldi1EnAWnunMWaet/lg4IPl75cBt5NMAgcDP7NJi+81xswyxmxmrX2p6RYqjJcxzU9kMW3atNTD1XkCN954Y6DivspkMi6brl7y6f9WD0BxL8oLtmbNGq67btQCzm2BHug+lSyVSq4/5FMyALUy0PQglueo8zHKtXx3bTabbTlOIJ/Pu/NpA6ivkulFX/6+vr6+YEIVeaZiRG1nYVJjjLuWVkW0WgnV+Smbielo1jA4R73YLwNzyt/nAsvVcSvK21IwxhxrjFlijFmycuXKJpsRERHRKlo2DFprrTGmYQuItfYiklLmLFiwwNZaRaYzwQqFHi+Di1xLswCZWSWabMstt+SGG25wbfIRkpZ+uXVrrVtrIG7Gxx57LMUEfPrZLmhDn/Sl3HtIkq1evRpI2MqMGTOauqbcy8yZM51U0y5ZXy0RtIMJQGIcBDjuuOOAJMDrzDPPBCqSPdTPwlDmzZvHVVddBVRKzRWLRV566aWq+2vns9JMTZ9fnpV87r333ik1ZiKYwCvGmM0Ayp+vlre/CGyhjtu8vC0iIqJD0SwTuB44Gjiz/Hmd2v5lY8wVJAbBVY3aA3xJKpJJSwOZuQuFQltYgW/MCYUSi/SaOXOm0/+alYohTKSh0A9jhmqDnL/qUfqllbBhOde0adPYYYcdAHjiiSeAsDtV2taO51ssFh27u+aaawDYYostOOuss4DqPA++QVDalc1m2X777QEQ9VUzGDl+k002AeC1115rud2ZTMaNQxnzb731lpPyn/jEJ4Ak6cu4GgaNMZeTGAE3McasAE4mefl/bYz5PPA8cHj58JuAA4FlwDrgs2NtiPhVxXAj/lkZgGvXrnWFMc8++2wATj75ZJfRpxWEvALyUmqrOcCLL77orP7txGTEDfT29qZ838aYqog4qDyDZvMLamy66aYudffHPvYx1w7JmycTq0QpttvKLmOor6+vyisA4QlHT0ZbbbUVAEuXLgWS2IA999wTwBmL5eVvR3xDX1+fEzgyGfX29vLxj38cgIsvvhiozkDUzDqPsXgHjqixa9/AsRb40pivHhERMenomIhBiVn3jWgyS8+YMcNJCcl9166SVj71y2QyvPXWW1XHyEw7ffr0cVEHJnItgdDJUKTZjBkzUhJMSqbtscce3HvvvU1dU/p41apVrkz47NmzgWpDrF/6vB3Q6oZE9mWz2VSUonb5yfPWElXuQfJeFotFHnjgAQCX489nT81AriPMAypjbc6cOVxyySWptrXCJOPagYiILkfHMAFIJJTonb6erl0eErN/wQUXBOPQG0Wt1FYasoLusccea0sKKR8TaRjU0XlyXbENrFmzJqUbC0vRZdQ1ahnvQvHz+j6lslGobTrZSTvcbtIOsTXobYJcLpcKxNH5B3x2YIzhPe95D1CxE0idjFWrVqX6dvr06akgNB2JuM022wCV/l66dKkL1Jo7d647l7AavV7F76OJcBFGRESsJ+gIJqCDIvwc/SHITHzkkUc6Cf3JT36y6ev71YMgLZklLFi3sRX4hT1HRkZSqbh1OnA/H3870n/D2JKJyrWHhobcdcUrI3o9VEJyRT/VOqteFVgvUayvS7cr+EaemfStzh2gpaY8j5B3yLcX6PO+973vBeChhx4CEh1eQsOFub7xxhtum5y3p6fHja37778fqHjGoMIABKFwdG1LawYdMQmEoKPJIDwp9Pb2cuihhwLwm9/8BoC/+Zu/cSqFXv4r0C8zJEYp2S/7QoYqvTBIBoF+WI1A34t20flUWB74eOaqa+Tcw8PDrq+kj/VyZL0NwpS0E5KpQNJGPzZBLwTzjcU6ulJPnHI/8rt3vvOdbt8rr7wCVGIHQmNYT3p6fQpUu/50O/w2hlSmsRRbFUR1ICKiy9ERTECvlgrt05+1IIEnV199NZ/61KeAMJUU9542AMlxwgCGh4cdE/HpYalUapoBaMj9SFoqbXjyj5ns+gqCYrGYWvKbzWZT0j20NFsfE6LV4w1//YG11gX2iEoTCrCRsTE0NOSCxPRxWmrr/3O5nKP+GjLG9KpTXT9A7wOcq1qCnLLZbIoJhIq2RsNgRETEmNERTKAdEN1q4cKFLvmFzPCiw22wwQZOigvz6OnpcVJeDFt6dhZpJce0kntfw080Yq1NrR33awLA5KS01hBD7BtvvAEk/SeuPpFgYkcRluND7nMimYBf8BTgc5/7HIBbHagDiHzm0N/fn9LPjTGpxCTazuCfI5fLuT6S56jdpCLt9blkDOtx4tc/GBoaasnFvN5MAtqQs/feewOVF8033mgUCgU3SeicgXI+eWjaut2I0aUWpG2/+tWvgOp8/360pN432aqBHy35xhtvuO/Sj35GJcBlGB4YGGjbRDpW1Mr3J0vCDz/8cLdNnocYOTX1DqmmtXIuGmPcmBHBozMoSXu0d8WfZOQ3+pqh2gjFYnHManMIUR2IiOhymMmWLJAkFZFyT81CZl1N5f17y2QywaXKfooobZgRaiuzuWYJAmNMasYObdNGJPn+jne8A0hooU8fdVlqURHancOuUfhSKFRu7dFHHwWSxB3STvGjt2PVZ6PQeRM1OwlR/4ULFwKklh4LVW8EQvX1cmA5r4yJNWvWpIqkCqPaaqut3JiU8vB67Mm5pk2bljIih5iBMWaptXaB387IBCIiuhzrjU0gZDwL1QLwXSk6yaUwgL/85S9u5vfdgVpqaN1NZm/tShQpKIawxYsXA4kUldVsco6+vj4X1+5HMHaSYdCvqhSq3yjJNzoF+plpHd+PIgT4r//6r6pt++23n/ud3KuOoJRt8in9MzQ05M4h0nzatGnuu9gc9Hj0C9fecMMNjkGJQVYbBnXCUz/KtJH6EOuNOqBDVmtRouHh4bphyfISTps2zXW0qAOhirWyxHZwcND9Vh7GW2+95QaEtGfRokVAMsn42XW1IaneMxmvvIPNQoc0hyaGToEfRapfJv3ytbN/Q+qGCJXQkmlRDV99NcnW19PT44qmSMzBjjvumDJ064xIAu3BEER1ICIiIoj1Rh0ISXj5ro2G9VwpOkmIpnVQmc1XrlzpJN4BBxwAJJlsxUijc+T56x+0ehLK7iv34DOIkPFtshGi0nK/OjVcKJ/hRCOTyaTSb2kKHSpE2w745+rr60sxgGw269oky4ZlfA0MDHDQQQdVtfvSSy/lox/9KFBJaaYjOaOLMCIiomGsN0xAx3Nr3R6qV1v5encoUm/BggWuBJfokmJzmD17tjPqhdxNepv81p/9c7lcKmBGux59/U7/X69k2kQi5KLUEZcQ1lUnA9ogHJKU48VS/HuX/hnt2iGbipzr6KOP5vvf/z5QST+2xx57VLmwgYaYQWQCERFdjvWGCUhlmeeee84FpIjeKhV0SqWSc9HIrDwyMpIqwa0DWvzZW+cf0PBjx/V5fYyMjKTSXeuAIB+hUumdiHYk2RxvdHL/heCHGRtjOP7444GKO/pb3/oWn/70p4EkQKtRdMwkUCwWsdYGM8DKpx95l8lk+NrXvgZUYr51jrzxWIo7mvtL0/RaL4OmgI2+MJNpYIuYWOTz+ZSRuKenJ1UX4pRTTmGvvfYC4O1vf3vD14nqQEREl6MjmIAsu9TGsXquPNn2z//8z66gp18nQM4bETFVUSgUUnkQh4aGUu/GwMCAMz6Pi4vQGHOJMeZVY8wjatvZxpgnjDF/NMb8pzFmltp3kjFmmTHmSWPM/g23KCIiYkIxFnXgUuCj3rbFwA7W2vcCfwJOAjDGbAcsArYv/+YCY8yYiqL5q/PEBiB/YmgbHh52yT4feeQRVq1a5Vx2ERHrG8S9mcvlnGtZvktB1N12243Zs2dX5SZoBGOpRXinMWaet+1m9e+9wGHl7wcDV1hrh4BnjTHLgF2Be0a7jkROieHLXwQUSuqgo/J8A0rE+otaSUImY13FWIzPkmFp3bp1VclpoDqLccgo7ntcdKzLbrvtBsB5553nFm2J0VCX1BsN7TAMfg74r/L3ucBytW9FeVsKxphjjTFLjDFL2lHGOSIiojm0ZBg0xvwLMAL8stHfWmsvAi6CZBWhL+V9F6GOPpPZrbe3t6sZQK18eLWOaYfLNCRt/TUPOofgeLg0Q4VDZXvosx3MQEtUHY1XK0O00HaoGK1LpVIqv6J+FqG1Jn6cwPz5892agRNPPBGAnXfeuaouht+O0dD0JGCMOQZYCOxrK3fyIrCFOmzz8raIiIgORVOTgDHmo8A3gL2ttevUruuBXxljvge8A9gauL+Rc/uBPqGEEOeffz6QFGysJ2lCUssvNKl1sqmE0VxBvtRvV2FPX8rqegl+abVcLleV+AIS3bbVdpRKJcc+9MpMP7GGltjNPmPdfj/ph84/4F87VFqvv78/lZhEIxSIJhWNJNfAqaeeyj777FN1vF5FKO1tJJnrqJOAMeZy4IPAJsaYFcDJJN6AXmBx+eHea609zlr7qDHm18BjJGrCl6y1Y+KDPhUKPTTZJgaRLbbYguXLl0s7U8dJh0iH6wepE2H4Od6mAkIP2agyWqGsSu2Ar6aJhVoj9OKEcvw1itAy4Hq0139pRztvCHqCq3cev1Kwvk+ddUiWTrvAQQAABodJREFUq8viMl28RRckkX1nnnkmUFm2XiwWUyHqWm1oRA1wbR/tAGvtEYHNF9c5/gzgjDG3ICIiYlLRERGDkJZcoaKWcsz73/9+AE4//XSXiumSSy4B4M4773Qzr6alUFmfAJOfq69VhGb/UFKMkJGpFWksEkyzJ5+ah2itPNe5c+e6Ap3yu76+PvddZ+aFxPgrlX6fe+45IHmOci25p3322Yedd97Z/QYqz72np6eq2Eytbfl8PrVN+q+vr88tTZc29vb2uuP0tfx70lmjfQmt1SM9TiF5rj7D0G0K5SccS8Stj7h2ICKiy9ExTEDgz2gycxYKhSrdHmDPPfd039/97ncDyXqC//7v/wbSs3lIAnZiUsyxQEt9HWzipy3TDKsddgG/qo7O2iuf0p4ZM2aksjWffvrpVRV/xoKLL060zzvuuANIJLDPGI444gh23XVXoPYYagb6XL77TWeq9mP8a7VDxptvyIPKWNTGa30tSJa2y2+lD4aHh1NVqxpBZAIREV2Ojkg5vvPOO9t77rmnyuXiB8KE8v3ncrmUxHvmmWdc5Zdzzz0XgBtvvBFI9FjfNTOa+8jX9caqX/b09FTVFIBqfVfrvLJPf9f7crmc+x66ppbEctyVV14JwC233OL6p53PWtdXOOusswD40pe+VNUefc1GdNRa0L/1vQN6Xytluutdu16KsrGeYzJRK+V4x0wCd999d1VH+/SqVCqlssPq4qD6d3LcihUrAKoyDflVibPZrHtJ9QtWrz5BJ6BUKqXapnPpPfzwwwA8++yzQPU96cnIn3jy+XyqP0KTjJ6gJrrAaERziHUHIiIigugIJmCMWQmsBTphJdEmxHZoxHZUYyq3413W2k39jR0xCQAYY5aEqEpsR2xHbMf4tiOqAxERXY44CUREdDk6aRK4aLIbUEZsRzViO6qx3rWjY2wCERERk4NOYgIRERGTgDgJRER0OTpiEjDGfLRcp2CZMebECbrmFsaY24wxjxljHjXGfKW8fSNjzGJjzFPlz+byODfenqwx5kFjzA3l/7c0xtxX7pMrjTE9o52jDW2YZYz5j3JNiceNMXtMRn8YY75afiaPGGMuN8b0TVR/mHCdjWAfmATnl9v0R2PM/HFux/jU+5CMKZP1B2SBp4GtgB7gf4HtJuC6mwHzy99nktRP2A44CzixvP1E4LsT1A9fA34F3FD+/9fAovL3HwJfmIA2XAb8Xfl7DzBrovuDJDv1s0C/6odjJqo/gA8A84FH1LZgHwAHkmTaNsDuwH3j3I6PALny9++qdmxXfm96gS3L71N2zNca74E1hpvdA/it+v8k4KRJaMd1wH7Ak8Bm5W2bAU9OwLU3B34H7APcUB5Ur6kHXtVH49SGDcsvn/G2T2h/UElbvxHJUvcbgP0nsj+Aed7LF+wD4EfAEaHjxqMd3r5DgF+Wv1e9M8BvgT3Gep1OUAfGXKtgvGCS4irvA+4D5lhrXyrvehmYMwFN+H8kiVtl+dvGwJvWWslnPRF9siWwEvhpWS35iTFmOhPcH9baF4FzgBeAl4BVwFImvj80avXBZI7dpup9hNAJk8CkwhgzA7ga+Edr7Wq9zybT6rj6UI0xC4FXrbVLx/M6Y0COhH5eaK19H8lajir7zAT1x2ySSlZbkmSsnk66DN6kYSL6YDSYFup9hNAJk8Ck1SowxuRJJoBfWmuvKW9+xRizWXn/ZsCr49yMPYGDjDHPAVeQqATnAbOMMZL5aSL6ZAWwwlp7X/n//yCZFCa6Pz4MPGutXWmtLQDXkPTRRPeHRq0+mPCxayr1Pj5VnpBabkcnTAJ/ALYuW397SAqaXj/eFzXJYvyLgcettd9Tu64Hji5/P5rEVjBusNaeZK3d3Fo7j+Teb7XWfgq4jUqNx4lox8vAcmPMNuVN+5Kkjp/Q/iBRA3Y3xkwrPyNpx4T2h4dafXA9cFTZS7A7sEqpDW2HqdT7OMim630sMsb0GmO2pNF6H+Np5GnAAHIgiXX+aeBfJuia/4eE1v0ReKj8dyCJPv474CngFmCjCeyHD1LxDmxVfpDLgKuA3gm4/k7AknKfXAvMnoz+AE4FngAeAX5OYvWekP4ALiexRRRI2NHna/UBiQH3B+Vx+zCwYJzbsYxE95fx+kN1/L+U2/EkcEAj14phwxERXY5OUAciIiImEXESiIjocsRJICKiyxEngYiILkecBCIiuhxxEoiI6HLESSAiosvx/wEeQuPnWyIqngAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "plt.imshow(imgs[0, :, :, 0], cmap=\"gray\")\n",
    "classes[np.argsort(labels[0, :])[-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型训练代码-模型构建\n",
    "\n",
    "TF V2.x 中使用动态图来构建模型，同时由于使用了 keras的 API，因此模型构建比较简单了。这里演示的是使用 class的方式构建模型，对于简单模型，还可以直接使用 Sequential 进行构建。\n",
    "\n",
    "这里的复杂模型也是用 Sequential 的简单模型进行的叠加。\n",
    "\n",
    "> 这里构建的是VGG模型，关于VGG模型的更多细节请参考 1409.1556。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyModel(tf.keras.Model):\n",
    "    def __init__(self):\n",
    "        super(MyModel, self).__init__()\n",
    "        # 模型有两个主要部分，特征提取层和分类器\n",
    "\n",
    "        # 这里是特征提取层\n",
    "        self.feature = tf.keras.models.Sequential()\n",
    "        self.feature.add(self.conv(64))\n",
    "        self.feature.add(self.conv(64, add_pooling=True))\n",
    "\n",
    "        self.feature.add(self.conv(128))\n",
    "        self.feature.add(self.conv(128, add_pooling=True))\n",
    "\n",
    "        self.feature.add(self.conv(256))\n",
    "        self.feature.add(self.conv(256))\n",
    "        self.feature.add(self.conv(256, add_pooling=True))\n",
    "\n",
    "        self.feature.add(self.conv(512))\n",
    "        self.feature.add(self.conv(512))\n",
    "        self.feature.add(self.conv(512, add_pooling=True))\n",
    "\n",
    "        self.feature.add(self.conv(512))\n",
    "        self.feature.add(self.conv(512))\n",
    "        self.feature.add(self.conv(512, add_pooling=True))\n",
    "\n",
    "        self.feature.add(tf.keras.layers.GlobalAveragePooling2D())\n",
    "\n",
    "        self.feature.add(tf.keras.layers.Dense(4096, activation=\"relu\"))\n",
    "        self.feature.add(tf.keras.layers.BatchNormalization())\n",
    "        self.feature.add(tf.keras.layers.Dense(4096, activation=\"relu\"))\n",
    "        self.feature.add(tf.keras.layers.BatchNormalization())\n",
    "\n",
    "        self.feature.add(tf.keras.layers.Dropout(0.5))\n",
    "\n",
    "        # 这个简单的机构是分类器\n",
    "        self.pred = tf.keras.layers.Dense(100)\n",
    "\n",
    "    def conv(self, filters, add_pooling=False):\n",
    "        # 模型大量使用重复模块构建，\n",
    "        # 这里将重复模块提取出来，简化模型构建过程\n",
    "        model = tf.keras.models.Sequential(\n",
    "            [\n",
    "                tf.keras.layers.Conv2D(filters, 3, padding=\"same\", activation=\"relu\"),\n",
    "                tf.keras.layers.BatchNormalization(),\n",
    "            ]\n",
    "        )\n",
    "        if add_pooling:\n",
    "            model.add(\n",
    "                tf.keras.layers.MaxPool2D(\n",
    "                    pool_size=(2, 2), strides=None, padding=\"same\"\n",
    "                )\n",
    "            )\n",
    "        return model\n",
    "\n",
    "    def call(self, x):\n",
    "        # call 用来定义模型各个结构之间的运算关系\n",
    "\n",
    "        x = self.feature(x)\n",
    "        return self.pred(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，上述模型定义中，仅仅关注当前模块的参数即可，模块的输入及输出的关系由框架自动推算得到，节省了很多精力。\n",
    "\n",
    "> 这跟 TF 1.x 有很大不同，也跟 pyTorch有很大不同。mxnet 的 gluon api 跟这里的操作是类似的。\n",
    "\n",
    "实例化一个模型看看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"my_model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "sequential (Sequential)      multiple                  33645760  \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              multiple                  409700    \n",
      "=================================================================\n",
      "Total params: 34,055,460\n",
      "Trainable params: 34,030,628\n",
      "Non-trainable params: 24,832\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = MyModel()\n",
    "\n",
    "model.build(input_shape=(None, 128, 128, 1))\n",
    "# 这里的build，是因为模型构建的时候，并没有指定输入数据的尺寸，\n",
    "# 因此要查看模型的一些数据，需要告知模型的输入数据尺寸，框架据\n",
    "# 此推断模型内部各模块参数。\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "build 和 summary 仅仅用来查看模型数据，对于模型训练不是必须的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型训练代码-训练相关部分\n",
    "要训练模型，我们还需要定义损失，优化器等，同时为了方便训练过程对模型进行验证，我们还需要定义一些性能指标。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_object = tf.keras.losses.CategoricalCrossentropy(from_logits=True)\n",
    "optimizer = tf.keras.optimizers.Adam()  # 优化器有些参数可以设置\n",
    "train_loss = tf.keras.metrics.Mean(name=\"train_loss\")\n",
    "train_accuracy = tf.keras.metrics.CategoricalAccuracy(name=\"train_accuracy\")\n",
    "\n",
    "val_loss = tf.keras.metrics.Mean(name=\"val_loss\")\n",
    "val_accuracy = tf.keras.metrics.CategoricalAccuracy(name=\"val_accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TF V1.x 中，模型的训练是需要启动一个 Session 的，而在 TF V2.x中，这个 Session的操作被隐藏了起来，取而代之的是 tf.function。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def train_step(imgs, labels):\n",
    "    with tf.GradientTape() as tape:\n",
    "        # tape 用来追踪整个计算图，并记录梯度\n",
    "        preds = model(imgs, training=True)\n",
    "        loss = loss_object(labels, preds)\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "\n",
    "    train_loss(loss)\n",
    "    train_accuracy(labels, preds)\n",
    "\n",
    "\n",
    "@tf.function\n",
    "def val_step(imgs, labels):\n",
    "    # 验证的时候，我们不需要进行梯度更新，\n",
    "    # 也就不需要使用tape\n",
    "    preds = model(imgs, training=True)\n",
    "    loss = loss_object(labels, preds)\n",
    "\n",
    "    val_loss(loss)\n",
    "    val_accuracy(labels, preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time  # 模型训练的过程中手动追踪一下模型的训练速度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为模型整个训练过程一般是一个循环往复的过程，所以经常性的保存重启模型训练中间过程是有必要的，这里使用 checkpoint 来保存模型中间训练结果，TF 整个系列对 checkpoint 的处理都很方便，这个目前是其他框架有欠缺的部分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "ckpt = tf.train.get_checkpoint_state(\".\")\n",
    "# 检查 checkpoint 是否存在\n",
    "if ckpt:\n",
    "    # 如果存在，则加载 checkpoint\n",
    "    model.load_weights(ckpt.model_checkpoint_path)\n",
    "    # 这里是一个比较生硬的方式，其实还可以观察之前训练的过程，\n",
    "    # 手动选择准确率最高的某次 checkpoint 进行加载。\n",
    "    print(\"model lodaded\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 0 Loss 7.023247718811035, Acc 1.0854661464691162, Val Loss 7.03449010848999, Val Acc 1.0532591342926025\n",
      "Speed train 333.99072580901236imgs/s val 855.2596907200536imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 1 Loss 6.777282238006592, Acc 1.0226234197616577, Val Loss 6.941394805908203, Val Acc 0.9141494631767273\n",
      "Speed train 327.1325337516975imgs/s val 916.1226758458906imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 2 Loss 6.359256267547607, Acc 1.174017310142517, Val Loss 5.80662727355957, Val Acc 1.2321144342422485\n",
      "Speed train 324.335567399578imgs/s val 962.8157509235044imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 3 Loss 5.362932205200195, Acc 1.388254165649414, Val Loss 5.001005172729492, Val Acc 1.5699522495269775\n",
      "Speed train 323.86471413694295imgs/s val 930.7858136706466imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 4 Loss 4.838431358337402, Acc 2.4737203121185303, Val Loss 4.622710704803467, Val Acc 3.5572338104248047\n",
      "Speed train 322.9755351536125imgs/s val 923.726279512034imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 5 Loss 4.364129066467285, Acc 5.950068473815918, Val Loss 3.7800049781799316, Val Acc 11.54610538482666\n",
      "Speed train 323.1431222660079imgs/s val 943.7244951999392imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 6 Loss 3.1959731578826904, Acc 21.5322208404541, Val Loss 2.499628782272339, Val Acc 36.08903121948242\n",
      "Speed train 322.8869991116342imgs/s val 930.8482939234336imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 7 Loss 2.12115216255188, Acc 46.052330017089844, Val Loss 1.6911603212356567, Val Acc 56.79650115966797\n",
      "Speed train 321.866934720262imgs/s val 936.5390705295545imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 8 Loss 1.473148226737976, Acc 61.46023941040039, Val Loss 1.1917930841445923, Val Acc 69.67408752441406\n",
      "Speed train 321.0786097004539imgs/s val 952.8346292198817imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 9 Loss 1.1069483757019043, Acc 70.93235778808594, Val Loss 1.013684868812561, Val Acc 73.92686462402344\n",
      "Speed train 319.400476897427imgs/s val 923.3046830012753imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 10 Loss 0.8779520988464355, Acc 76.97097778320312, Val Loss 0.8483538031578064, Val Acc 78.67646789550781\n",
      "Speed train 309.2933807436792imgs/s val 900.1829360283824imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 11 Loss 0.7251728773117065, Acc 80.77296447753906, Val Loss 0.7506675124168396, Val Acc 81.0214614868164\n",
      "Speed train 301.2632010228335imgs/s val 898.4208917080757imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 12 Loss 0.6008656024932861, Acc 83.94367218017578, Val Loss 0.7291795015335083, Val Acc 81.25993347167969\n",
      "Speed train 297.387641228917imgs/s val 860.7664859874725imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 13 Loss 0.5257374048233032, Acc 85.77182006835938, Val Loss 0.6249831318855286, Val Acc 83.94276428222656\n",
      "Speed train 294.72389001523715imgs/s val 870.8659186105319imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 14 Loss 0.47265708446502686, Acc 87.06867218017578, Val Loss 0.628078043460846, Val Acc 84.91653442382812\n",
      "Speed train 292.79691865600137imgs/s val 885.4427240527049imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 15 Loss 0.41973787546157837, Acc 88.53118896484375, Val Loss 0.5823920965194702, Val Acc 86.40699768066406\n",
      "Speed train 290.9758017995805imgs/s val 861.7976330609051imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 16 Loss 0.35504868626594543, Acc 89.89087677001953, Val Loss 0.5734841823577881, Val Acc 86.44673919677734\n",
      "Speed train 287.8656709680878imgs/s val 843.401053133528imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 17 Loss 0.32744738459587097, Acc 90.64213562011719, Val Loss 0.5623900294303894, Val Acc 87.400634765625\n",
      "Speed train 286.3891225531947imgs/s val 850.7197864862696imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 18 Loss 0.26646262407302856, Acc 92.30461883544922, Val Loss 0.5681359767913818, Val Acc 87.10254669189453\n",
      "Speed train 293.50211639303797imgs/s val 890.6136287690091imgs/s\n",
      "Found 35008 images belonging to 100 classes.\n",
      "Found 5000 images belonging to 100 classes.\n",
      "Epoch 19 Loss 0.3041706383228302, Acc 91.7533187866211, Val Loss 0.9558600187301636, Val Acc 79.47138214111328\n",
      "Speed train 314.4602859964906imgs/s val 918.107688418758imgs/s\n"
     ]
    }
   ],
   "source": [
    "EPOCHS = 20\n",
    "for epoch in range(EPOCHS):\n",
    "    # 验证数据都是针对整个 epoch 的，\n",
    "    # 所以每个 epoch 之间要对这些数据初始化一下。\n",
    "    train_loss.reset_states()\n",
    "    train_accuracy.reset_states()\n",
    "    val_loss.reset_states()\n",
    "    val_accuracy.reset_states()\n",
    "    total_trained = 0\n",
    "    total_valed = 0\n",
    "\n",
    "    start = time.time()\n",
    "\n",
    "    for imgs, labels in img_gen_train.flow_from_directory(\n",
    "        \"worddata/train/\",\n",
    "        target_size=(128, 128),\n",
    "        color_mode=\"grayscale\",\n",
    "        classes=classes,\n",
    "        batch_size=batch_size,\n",
    "        shuffle=True,\n",
    "    ):\n",
    "        # 之前生成的 img_train 是个 generator，而且它不会自动重启和结束，\n",
    "        # 一旦启动，只要不手动结束，它会一直无限循环输出数据，因此这里手动处\n",
    "        # 理一下数据的生成，并做一个计数。\n",
    "\n",
    "        train_step(imgs, labels)\n",
    "        total_trained += imgs.shape[0]\n",
    "        if total_trained > 35000:\n",
    "            break\n",
    "    period = time.time() - start\n",
    "    train_samples_per_second = total_trained / period\n",
    "\n",
    "    start = time.time()\n",
    "    for imgs, labels in img_gen_val.flow_from_directory(\n",
    "        \"worddata/validation/\",\n",
    "        target_size=(128, 128),\n",
    "        color_mode=\"grayscale\",\n",
    "        classes=classes,\n",
    "        batch_size=batch_size,\n",
    "    ):\n",
    "        val_step(imgs, labels)\n",
    "        total_valed += imgs.shape[0]\n",
    "        if total_valed > 5000:\n",
    "            break\n",
    "    period = time.time() - start\n",
    "    val_samples_per_second = total_valed / period\n",
    "\n",
    "    print(\n",
    "        \"Epoch {} Loss {}, Acc {}, Val Loss {}, Val Acc {}\".format(\n",
    "            epoch,\n",
    "            train_loss.result(),\n",
    "            train_accuracy.result() * 100,\n",
    "            val_loss.result(),\n",
    "            val_accuracy.result() * 100,\n",
    "        )\n",
    "    )\n",
    "    print(\n",
    "        \"Speed train {}imgs/s val {}imgs/s\".format(\n",
    "            train_samples_per_second, val_samples_per_second\n",
    "        )\n",
    "    )\n",
    "    model.save_weights(\"model-{:04d}.ckpt\".format(epoch))\n",
    "    # 每个 epoch 保存一下模型，需要注意每次\n",
    "    # 保存要用一个不同的名字，不然会导致覆盖，\n",
    "    # 同时还要关注一下磁盘空间占用，防止太多\n",
    "    # chekcpoint 占满磁盘空间导致错误。\n",
    "    # 注意这个 API 调用每次都会生成数个文件，\n",
    "    # 其中 checkpoint 文件用来记录每次的文\n",
    "    # 件路径，其他文件则存储模型数据和索引信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 一些技巧\n",
    "\n",
    "因为这里定义的模型比较大，同时训练的数据也比较多，每个 epoch 用时较长，因此，如果代码有 bug 的话，经过一次 epoch 再去 debug 效率比较低。\n",
    "\n",
    "这种情况下，我们使用的数据生成过程又是自己手动指定数据数量的，因此可以尝试缩减模型规模，定义小一些的数据集来快速验证代码。在这个例子里，我们可以通过注释模型中的卷积和全连接层的代码来缩减模型尺寸，通过修改训练循环里面的数据数量来缩减数据数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
